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Abstract: We consider a general honest homogeneous continuous-time Markov process with 
restarts. The process is forced to restart from a given distribution at time moments generated by 
an independent Poisson process. The motivation to study such processes comes from modeling 
human and animal mobility patterns, restart processes in communication protocols, and from 
application of restarting random walks in information retrieval. We provide a connection between 
the transition probability functions of the original Markov process and the modified process with 
restarts. We give closed-form expressions for the invariant probability measure of the modified 
process. When the process evolves on the Euclidean space there is also a closed-form expression for 
the moments of the modified process. We show that the modified process is always positive Harris 
recurrent and exponentially ergodic with the index equal to (or bigger than) the rate of restarts. 
Finally, we illustrate the general results by the standard and geometric Brownian motions. 

Key-words: Markov Processes with Restart, Positive Harris Recurrence, Exponential Ergodicity, 
Standard and Geometric Brownian Motions 
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Les Processus de Markov avec Redemarrage 



Resume : Nous considcrons un processus de Markov en temps continu qui 
est general, honnctc ct homogcnc. Lc processus est force a redemarrer a par- 
tir d'une distribution donnee a des moments de temps generes par un proces- 
sus independant de Poisson. La motivation pour etudier ce type de processus 
vient de la modelisation de la mobilite humaine et animale, des procedures 
avec le redemarrage dans les protocoles de communication et de l'application 
des marches aleatoires avec le redemarrage en recherche d'information. Nous 
fournissons une connexion entre les fonctions de probabilite de transition du pro- 
cessus de Markov originalc et du processus modifie avec le redemarrage. Nous 
donnons des expressions explicites de la mesure invariante du processus modifie. 
Lorsque le processus evolue dans l'espace Euclidien, il y a egalement une expres- 
sion explicite pour les moments du processus modifie. Nous montrons que le 
processus modifie est a la fois Harris positif recurrent et ergodique exponentiel 
avec l'indice egal a (ou plus grand que) le taux de redemarrage. Enfin, nous 
illustrons les resultats generaux par les cas classiques de mouvements Browniens 
standard et geometrique. 

Mots-cles : Processus de Markov avec Redemarrage, Positive Harris Recur- 
rence, Ergodicite Exponentiel, Mouvements Browniens Standard et Geometrique 
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1 Introduction 

Many phenomena in nature and technology can be modeled by Markov pro- 
cesses which restart from time to time. Human and animal movements can be 
modeled by Markov processes that restart from some locations [H [05] . Facing 
the congestion of the Internet traffic, the Internet users tend to restart their 
sessions [TTJ [TJ] or a protocol governing the rate of information transmission 
invokes restart routines (i.e., Slow-Start routine in the Internet Transmission 
Control Protocol) |T5] - The celebrated PageRank algorithm [3 in information 
retrieval models the behavior of a random surfer, who decides to restart web 
surfing from time to time. The heat kernel PageRank [4 is a continuous-time 
analog of the original discrete-time algorithm. The restart policy is also used 
to speedup the Las Vegas type randomized algorithms [U [13] . 

Motivated by the above mentioned phenomena, we study a general honest 
homogeneous continuous-time Markov process with restarts. We assume that 
the process of restarts is modeled by an independent Poisson process. In the next 
Section 2 we derive a formula which makes a connection between the transition 
probability functions of the original Markov process and the Markov process 
modified by restarts. As a corollary we obtain a closed-form expression for the 
invariant probability measure of the modified process. We also show that the 
modified process is always positive Harris recurrent and exponentially ergodic 
with the index equal to (or bigger than) the rate of restarts. In Section 3 
we provide bounds for the moments of the modified process and investigate 
the limiting behaviour of the modified process when the rate of restarts goes 
to zero. We conclude the paper with Section 4 where we consider the restart 
modifications of the two classical Markov processes: the standard Brownian 
motion with drift and the geometric Brownian motion. It is very curious to 
observe that in the case of the geometric Brownian motion the number of finite 
moments of the modified process depends on the rate of restarts. Thus, even 
a small change in the value of the restart rate can lead to dramatic changes of 
the properties of the geometric Brownian motion with restarts. 



2 Main results 

In accordance with [12] , we consider a homogeneous continuous-time Markov 
process {X(t) : t <E [0, oo)} in a Borel space (E,£) characterized by the initial 
distribution 7(-), and the honest transition (probability) function P(t, x, dy), 
satisfying the following properties: 

1. P(t,x, •) is a probability measure on E, 

2. P(o,x,r) = i{x e r}, 

3. for each fixed r G £, t 6 [0, oo), P{t, x, V) is a jointly measurable function 
with respect to (i, x) € [0, oo) x E, 
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4. and the Chapman-Kolmogorov equation takes place 

P(t + s,x,T)= [ P(s,y,T)P(t,x,dy). 

JE 

Throughout this paper, all the processes are from the same probability space 
(fi, J-, P). We further use the notations P x , E x , V ar x when the initial state of 
the concerned process is x £ E. 

The goal of the present work is to analyse a modification of the Markov 
process {X(t) : t £ [0,oo)} introduced in the above by forcing the process to 
restart with a given restart distribution ^(r) after an exponentially distributed 
random time, i.e., the process counting restarts represents a standard Poisson 
process with the rate, say, A > 0, independent of {X(t) : t £ [0,oo)}. In 
the following theorem we characterize the transition function of the modified 
Markov process, which is denoted as {X(t) : t £ [0,oo)}. 

Theorem 1 Let P(t, x, T) be the transition function of the Markov process 
{X(t) : t £ [0,oo)} taking values in a Borel space (E,£). Then, the modi- 
fied Markov process {X(t),t £ [0,oo)} that restarts from a distribution v after 
(independent) exponentially distributed random times with mean 1/A has the 
following transition function 

P„{t,x,T) = e~ xt P(t,x,T)+ ( f Xe- Xs P(s,y,T)dsiy(dy). (1) 

JE JO 

Proof: Denote by N(t) the number of restarts up to time t and by S the time 
elapsed since the time moment of the last restart before t. Then, we have 

p u (t, x, r) = P[{x{t) er}n {N{t) = o}|l(o) = x] 



+P[{x(t) £ r} n {N(t) > 0}\x{0) = x}. (2) 

The first term in the above equation is given by 

p[{x{t) £ r} n {N{t) = o}|l(o) = x} = e - xt p{t, x, r). 

Let us now calculate the second term in |2]). Note that the second term in |2]) 
can be expressed as 

P[{X(t) £ T} n {N(t) > 0}\X(0) = x] = f [ P(s,y,T)dF(s)v(dy), (3) 

JE JO 

where F(s) — P[S < s\N(t) > 0]. Let Xj, denote the fc-th restart moment 
from time zero. If N(t) = n > 0, then the restart times Ti,...,T n have the 
same distribution as the order statistics corresponding to n independent random 
variables uniformly distributed on the interval (0,t) |17j . and therefore, 

/ T \n 

P[T n <T\N(t)=n]={~) , r€[0,i], 7i=l,2,.... 
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Then, 



P[S < s\N(t) =n} = P[T n > t - s\N(t) = n] = 1 - 
and, consequently, 



t - s 
t 



, n = 1,2,..., 







1 - 





Hence, 

P[{x(t) e r} n {N(t) > 0}|x(0) = &] 

and the formula ([!]) follows. 



1 - e 



-As 



Ae- A5 P(s,y,r)dsz.(dy), 



□ 



Corollary 1 TTie measure 
9u(T) = 



f Xe- Xs P{s,y,T)dsu{dy) 

JE JO 



(4) 



is an invariant probability measure for P v {t, x, T). 

Proof: Since P(s,y,-) is a probability measure for all s and y, by the Vitali- 
Hahn-Saks theorem q v {-) is a probability measure as well. Let us show that it is 
indeed an invariant measure. It is enough to show this for v{dy) — l{w € c?y}, 
where w £ E is fixed, and in that case we also denote q v by q w and P v by P w 
for brevity. By Theorem [T] we obtain 



q w (dz)P w {t,z,T) = / q w {dz) e- M P(t,z,T)+ / Ae s P(s, w, T) ds 

JE I JO 

= e~ xt [ q w {dz)P(t 1 z,T)+ [ \e~ Xs P(s, w, T) ds. 

J E Jo 

Substituting into the above equation 



poo 

/ \ e - Xs p{s,w,r)ds 

Jo 



and using the Chapman-Kolmogorov equation yields 



q w (dz)P w (t,z,T) = e 



-XL 



Xe- Xs P(s + t,w,r)ds+ I \e-* s P(s,w,T)ds 



-As \ 



/OO pt 
\e- Xs ' 'p(s',w,T)ds' + J \e- Xs P{s,w,T)ds 



\e- Xs P(s,w,T) ds = q w {T), 
which concludes the proof of the corollary. 



□ 
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Remark 1 We note that alternatively formula can be rewritten 

g„(T)= / \R(y,T)v(dy), 



where R(y,T) = /q°° e Xs P(s,y,T) ds is the resolvent operator. 

Now we are ready to prove that the modified process is positive Harris re- 
current and exponentially ergodic with index A. Before this, let us remind the 
definitions. 

Recall that a homogeneous continuous-time Markov process {X(t), t € [0, oo)} 
is called Harris recurrent if there exists a non-trivial a-finite (recurrence) mea- 
sure [i on 8 such that for each x <E E, 

r r := inf{i > : X{t) e V} < oo, P x - a.s., 

whenever /j,(F) > 0, see [3 US]. 

Then, if a Harris recurrent process admits an invariant probability measure, 
it must be unique, and the process is further called positive Harris recurrent, 
see 0H51. 

A homogeneous continuous-time Markov process {X(t),t € [0, oo)} is called 
exponentially ergodic with index a if there exist a probability measure /x(-) 
on £ 1 a finite valued function M(-) on E, and a constant a > 0, satisfying 
\\P(t,x, ■) — ju(-)||tv < M(x)e~ at for every x £ E, where || • \\tv denotes the 
total variation norm, see [5]. 

Theorem 2 The modified Markov process {X{t),t € [0, oo)} is positive Harris 
recurrent and exponentially ergodic with index equal to [or bigger than) the rate 
of restarts A, and the following inequality takes place 

\q v (T)-P u (t,x,T)\ < e- At , VxeE, VTe£. (5) 

Proof: The modified process {X(t),t G [0, oo)} under consideration is Harris 
recurrent with the recurrence measure given by the restart distribution /i(-) = 
v( ). From Corollary 1 we conclude that {X(t),t G [0, oo)} is positive Harris 
recurrent with the unique invariant probability measure given by Q. Let us 
now prove We see 

\q v (T)-P v (t,x,T)\ = \ [ ( \e- Xs P(s,y,T)dsv{dy)-e- xt P{t,x,Y)\ 



p poo poo 

/ / \e- Xs P{s,y,T)dsv{dy)- \ e - Xs P(t,x,T) ds\ 

J E Jt Jt 

P(s, yi T)v(dy)~P(t,x,r) t 
f Xe- Xs \ f P(s,y,T)v(dy)-P(t,x,T)\ds< 



ds\ 
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Since \\P{t,x,-) - «„(-)||tv = 2sup re£ \P(t,x,T) - q v (T)\ 9, Appendix], it 
follows that {X(t),t € [0, oo)} is exponentially ergodic with index A and M(x) = 
2. 

□ 

3 Moments and limits 

In this section, we let the initial distribution of the original process be the Dirac 
measure concentrated at x € E, where E = R™. Then we write {X(t) : t > 0} 
as {(Xx(t), . . . , X n (t)) : t > 0}. Similar notations are introduced for the process 
{X(t) : t > 0}, too. Now consider the ith component process {Xi(t),t > 0}. 
From ([I]) we can obtain an expression for the moments 

E x [X*(t)] = e- Xt E x [X*(t)] + [ f \e- x *E y [Xl{s)] dsv(dy), (6) 

JE JO 

where and below in this section, we assume the involved interchange of the order 
of integrals is legal, which is the case, for example, when 

f f \e- Xs E y [\X?(s)\]dsv(dy) < oo. 

JE JO 

In turn, the equation ^ helps to establish the following bound. 

Proposition 1 Let the k-th moment of the original process be exponentially 
bounded from the above in time, i.e., 

E x {X^{t)]<c k ,i{x)e^\ (7) 

where Ck,i(-) is a measurable v-integrable function, and rjk.i < A is a constant. 
Then, 

lim sup < (8) 

t-s-oo A — T)k,i 

where c^,i — f E Ck,i(y)v(dy). If the k-th moment of the original process is 
uniformly bounded from the above (i.e., E x [X^(t)] < Ck,i € (—00,00)), so is the 
k-th moment of the modified process by the same bound (i.e., E x [X^(t)] < Ck.i)- 

Proof: The equation ^ yields 

E x [X-(t)} < e" A *c fe , l (x)e'"- t + / / Xe^c^e™-* 8 ds v(dy) 

Je Jo 

= e-^-'^CkAx) + (1 - e-^-^) f c Kl (y) V (dy), 

A - Vk,i Je 

which implies the first statement of the proposition. The second statement of 
the proposition also follows from the equation 
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Now let us investigate what happens when the parameter A goes to zero. 
Since J E P(s,y,T)v(dy) is a bounded measurable function with respect to s, 
according to [5J, we can conclude that the limiting discounting is equivalent 
to the time averaging. This fact and Theorem 1 in [7] lead to the following 
statement. 

Theorem 3 The existence of the limit lim^o q v (r) is equivalent to the exis- 
tence of the limit liuiT^oo^/T J Q J E P(s,y,T) v(dy) ds. If these limits exist, 
they are equal. In particular, if the original process {X(t) : t G [0, oo)} is posi- 
tive Harris recurrent, the limit lim^o 9i/(r) exists and is equal to the invariant 
probability measure of the original process. 

Remark 2 There could be cases when the limit lim^o (?„(•) exists for any prob- 
ability distribution v{-). However, similarly to the case of singularly perturbed 
Markov processes f^, the original process might not be ergodic. 

4 Examples 

Let us illustrate the general results with two examples. 
4.1 Brownian motion with drift 

As the first example let us consider the Brownian motion with drift \x and vari- 
ance coefficient a 2 on the real line E = K (see e.g., [T7]). The initial distribution 
of the original process is the Dirac measure concentrated at x G E. It can be 
described in the stochastic differential notation 



where W(t) is the standard Wiener process. The probability density function 
of the Brownian process has a closed form 



Here we assume that the process starts from zero. We observe that p(t, 0, z) 
does not converge to a proper probability density as t goes to infinity. If the 
modified process restarts also from zero, by formula (Til) we have 



dX{t) = fj,dt + adW(t), 






(9) 
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which has the well defined limiting probability density function 

Let us now consider that the original process starts with an arbitrarily fixed 
x £ E and restarts according to the distribution v{dy), which admits a finite 
second moment. Then we may calculate the first moment of the modified process 
by formula M with E x [X(t)] = x + fit 

E x [X(t)} = e~ xt (x + fxt) + [ [ Xe~ Xs (y + ^s)dsv{dy) 

J E JO 

= e~ Xt (x + fit) + [1 - e~ xt ] f yv(dy) + [1 - (1 + At)e~ At ]^. 



Thus, 



E x [X(t)}^ I yv(dy) + £ 



as t — > oo. 

Similarly, a direct calculation gives 

E x [X 2 (t)\ -> y + ^ + ^(^ + y 2 )^(dy), as t -> oo, 

and thus 

<> / /• \ 2 2 2 



Far x [X 2 (i)] -> J^y 2 v{dy) - \J^yv{dy)j + y + ^, as t -> oo. 
4.2 Geometric Brownian motion 

As the second example we consider the geometric Brownian motion (see e.g., 
[IT]), so that we take E = [0, oo). It can be described by the following stochastic 
differential equation (with the initial condition P(X(0) £ dy) = l{x £ dy}, 
where x £ E is fixed) 

dX(t) = (j,X(t)dt + aX(t)dW(t), 

where W(t) is the standard Wiener process. The probability density function 
of the geometric Brownian motion also has a closed form 

P (t x z) ^^exp (_ Mz)-H')-^-^/m a ) {1Q) 

which defines a log-normal distribution with mean 

E w [X(t)]=xef it , (11) 
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and variance 

Var x {X{t)\ = x 2 e 2 »\e aH - 1). (12) 
Indeed, for any fc = 1,2,..., it holds for this log-normal distribution that 

Next, for any fixed k — 1,2,..., we assume A — k{[i — ^-) — \k 2 (j 2 ^ 0, 
and v(dy) has a finite kth. moment. By using formula ([6]), we calculate the fcth 
moment of the modified process 



E x [X k {t)] = e -At x fe e fe(M-V)*+|fe d " "*+ / / Xe- Xs y k e k ^-^ s+ ^ kas dsv{dy) 

Je Jo 

= x k e -t(x-k ( ^4)-^^) + f Xyk f l ^ s{ x^-4)-^^) dsv{dy) 



Thus, when A > — ^) + \k 2 a 2 > 0, we have 
as t — > oo. 
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